System and method for providing content-aware persistent advertisements

ABSTRACT

A method and system of providing content-aware persistent advertisements comprises the steps of ingesting and collecting VDNA (Video DNA) fingerprints of media contents playing on various network-enabled terminals including Internet browsers, mobile devices, tablets, smart televisions and so on; sending the ingested fingerprints along with other information such as metadata, user&#39;s location, etc, to the content identification server via Internet or mobile networks, and selecting accurate advertisements according to the media content and relevant information, then finally pushing the advertisements back to user&#39;s terminal. With VDNA fingerprint identification, media contents are identified by content instead of metadata or other surrounding information, so that the result of identification can be considered genuine, and the chosen advertisements based on the media content can be persistent across multiple terminals playing the same media content.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention, which relates to a method and system forproviding content-aware persistent advertisements, comprises the stepsof extracting and collecting VDNA (Video DNA) fingerprints of mediacontents playing on various network-enabled terminals including internetbrowsers, mobile devices, tablets, smart televisions (TVs) and so on,sending the extracted fingerprints along with other information such asmetadata, user's location, etc, to the content identification server viainternet or mobile networks, and selecting accurate advertisementsaccording to the media content and relevant information, then finallypush the advertisements back to user's terminal. Specifically, thepresent invention relates to facilitating content-aware persistentadvertisements.

2. Description of the Related Art

The internet has become an ongoing emerging source that tends to expandmore and more. The growth of this particular media attracts theattention of advertisers as a more productive source to bring inconsumers. A clear advantage a consumer has with online advertisementsis that he or she has control over the item, choosing whether to checkit out or not. Online advertisements also can offer various forms ofanimation. In its most common use, the term online advertising comprisesall sorts of banner advertisement, e-mail advertising, in gameadvertising, and keyword advertising, on many social network platforms,which have received increased relevance. Web related advertising has avariety of sites to publicize and reach a niche audience to focus itsattention to a specific group. Research has proven that onlineadvertising has given results and has growing business revenue.

Deploying online advertisements along with various contents includingvideo, audio, images or texts, is a crucial strategy for content ownersto realize monetization on their contents. Online advertisements areseen very often on websites, free applications, online videos, etc. onterminals such as PCs (personal computers), smart TVs, mobile devices ortablets. There are several common ways in which online advertising ispurchased: CPM (Cost Per Mille), CPC (Cost Per Click), and CPA (Cost PerAction). Content owners apply these revenue models along with thecontents they possess.

Nowadays examples of online advertising include contextual advertisingon search engine results pages, banner ads, rich media ads, socialnetwork advertising, and so on, wherein contextual can be 1) textcontext, which is mostly used at present, advertisement engines useskeywords of the contents or the application or websites containing thecontents to determine appropriate advertisements, 2) metadata of thecontent, such as title, author, publish date of contents like videos,albums, 3) location context, some applications or websites are able toacquire users' location information, therefore the advertisements theydeploy can be location aware.

Contextual advertising is a form of targeted advertising foradvertisements appearing on websites or other media, such as contentdisplayed in mobile browsers. The advertisements themselves are selectedand served by automated systems based on the content displayed to theuser. A contextual advertising system scans the text of a website forkeywords and returns advertisements to the webpage based on what theuser is viewing. The advertisements may be displayed on the webpage oras pop-up ads. Contextual advertising is also used by search engines todisplay advertisements on their search results pages based on thekeywords in the user's query. Contextual advertising is a form oftargeted advertising in which the content of an ad is in directcorrelation to the content of the web page the user is viewing.Contextual advertising is also called “In-Text” advertising or“In-Context” technology. Contextual advertising has made a major impacton earnings of many websites. Because the advertisements are moretargeted, they are more likely to be clicked, thus generating revenuefor the owner of the website (and the server of the advertisement).There are several advertising agencies that help brands understand howcontextual advertising options affect their advertising plans. There arethree main components to online advertising:

-   -   1) creation—what the advertisement looks like    -   2) media planning—where the advertisements are to be run    -   3) media buying—how the advertisements are paid for

Contextual advertising replaces the media planning component. Instead ofhumans choosing placement options, that function is replaced withcomputers facilitating the placement across thousands of websites.

However, there are some disadvantages on the current methods ofcontextual advertisements: 1) the context information or the metadata ofthe content are usually not verified by content owner, they are mostlyedited by the website or application which distribute the content,therefore it is possible that the advertisements chosen for the contentbased on the contextual information are not relevant to the content ornot desired by content owners; 2) automatic delivery of content-awareadvertisements are not possible; 3) the current method of advertisementdistribution does not combine all retrievable information to select mostaccurate and user prone ads; 4) such methods cannot select accurateadvertisements for terminal devices when they are playing contents likevideos or audios, because those contents playing on terminal devices,whether they are online or locally saved, provide limited metadata orinformation. Moreover, the advertisements displaying on differentterminal devices are now-independent; it is not possible for contentproviders or advertising agents to apply proper business rules on thoseadvertisements, wherein the business rules, for example, could bedisplaying a serial of advertisements for the same media contentsplaying on different devices. There are not feasible in the currentmethods of contextual advertisements to determine accurately the mediacontents showing on the screens of different devices and to selectproper advertisements according to predefined business rules.

Ways to automatically identify the contents which users are watching onvarious terminals, and choose accurate online advertisements are hencedesirable, so that no or few human operations are involved in the wholeprocess. With the help of a mature media fingerprinting technology,given required content and metadata from content owners and users, thesystem is able to identify any number or format of media contents, andpush the most accurate advertisements which content owners and usersdesire.

All these and other introductions of the present invention will becomemuch clear when the drawings as well as the detailed descriptions aretaken into consideration.

SUMMARY OF THE INVENTION

An object of the present invention is to overcome at least some of thedrawbacks relating to the prior arts as mentioned above.

Conventional deployment of online content accompanied advertisements isbased on textual context of the metadata from the displaying content.Such metadata information is usually not verified by the content owner;they are mostly edited by the website or application which distributesthe content, so that it is not possible for content owners andadvertisement agents to deploy content-aware persistent advertisementsacross different terminals and platforms. However in the presentinvention, the content-aware persistent advertisement system isfacilitated with media content identification sub-system, which enablesswift and automatic online media identification and advertisementselection.

An object of the present invention is to automatically identify mediacontents playing on various devices and push relevant advertisementsaccording to the content and other information, such as a user'slocation, in a fast and accurate way. The present invention comprisessteps of extracting and collecting VDNA fingerprints of media contentsplaying on various network-enabled terminals including internetbrowsers, mobile devices, tablets, smart TVs and so on, sending theextracted fingerprints along with other information such as metadata,user's location, etc, to the content identification server via internetor mobile networks, and selecting accurate advertisements according tothe media content and relevant information, then finally push theadvertisements back to the user's terminal.

Extracting and collecting fingerprints out from media contents ondevices takes advantage of the rapidly increasing processing speed ofthe mobile devices nowadays to extract characteristic values of eachframe of image and audio from media contents, as is called “VDNA (VideoDNA)”, which are registered in VDDB (Video Digital Data-Base) forreference and query. Such process is similar to collecting and recordinghuman fingerprints. One of the remarkable uses of VDNA technology is torapidly and accurately identify media contents, so that it is possibleto select and push relevant advertisements in mobile networks.

Another characteristic of VDNA fingerprints is that it is very compact,so that it is feasible to transfer over mobile networks. Because someterminals may use mobile networks, which always have lower bandwidth,sending huge amounts of information of the media content to the contentprovide for identification is not realistic, while sending only metadataof the media content may not result in accurate selection ofadvertisements. Therefore extracting key characteristics of the mediacontents and sending the extracted fingerprints of the media contentsremits the mentioned disadvantages.

The VDNA fingerprint process is performed on terminals where mediacontents are playing on, therefore additional software components arerequired to install on these terminals such as plug-ins for internetbrowsers, dedicated APIs (application program interface) for mobiledevices and tablets or interfaces designed for smart TVs. These softwarecomponents help to collect fingerprints of the on play media contents aswell as other metadata information and user specific data. Such datawill be sent via internet or mobile networks to content identificationserver, where the media content can be identified.

Advertisement selection is based on the identified contents; therefore,media contents showing on different terminals are able to triggerpersistent advertisements or sets of relevant advertisements predefinedby content providers and ad agents under specific business rules. Userswho use the media contents have the options to choose what kinds of userspecific data to share. Advertisement selection also takes these userspecific data into account to generate advertisements most suitable forusers. Content provides and advertisement agents will predefine somebusiness rules for the choice of advertisements. These rules are crucialfor generating content-aware persistent advertisements. The terminologyof content-aware persistent advertisement is self-explanatory, whereascontent-aware means that the advertisement is selected according to therecognition result of the media content, which comes along from hecontent identification center. In addition, the advertisements are alsopersistent, whereas persistency can have multiple forms, such as:

-   -   1) geo-location persistency, meaning that for users with        terminal devices in a same range of locations, they should be        assigned with advertisements with specific location hints or        information, for example promotion of the nearest products which        appears in the recognized media content.    -   2) time persistency, meaning users with terminal devices in the        same duration of time, they should be assigned with        advertisements with specific date or time hints or information.        For example, products on sale on a specific date of a week.    -   3) environment persistency, meaning that for users with terminal        devices in the same environment, they should be assigned with        advertisements with specific environmental hints or information.        For example, with user's location the backend servers are able        to deduce the weather where the user is located. With such        information, proper environmental relevant advertisements can be        selected.    -   4) user preferences persistency, meaning users with terminal        devices who share similar preferences on media contents should        be assigned with advertisements with specific preference hints        or information. For example, similar artworks can be promoted        for users who are interested in the same kind of genre.

There are more forms of persistency besides the above list, and theycould also be combined together to generate more specific and properadvertisement selections. In summary, the present invention takesadvantage of the properties of computers, modern mobile devices andnetworks: high speed, automatic, huge capacity and persistent, andidentifies media contents in very high efficiency, making it possiblefor content providers and ad agents to automatically, accurately andrapidly push relevant persistent advertisements to the terminal.

In other aspect, the present invention also provides a system and a setof methods with features and advantages corresponding to those discussedabove.

All these and other introductions of the present invention will becomemuch clear when the drawings as well as the detailed descriptions aretaken into consideration.

BRIEF DESCRIPTION OF THE DRAWINGS

For the full understanding of the nature of the present invention,reference should be made to the following detailed descriptions with theaccompanying drawings in which:

FIG. 1 shows schematically a component diagram of each functional entityin the system according to the present invention.

FIG. 2 is a flow chart showing a number of steps of contentidentification in the aspect of terminal side.

FIG. 3 is a flow chart showing a number of steps of contentidentification in the aspect of server side.

FIG. 4 is a flow chart showing a number of steps of how the associatedadvertisements are selected.

Like reference numerals refer to like parts throughout the several viewsof the drawings.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention now will be described more fully hereinafter withreference to the accompanying drawings, in which some examples of theembodiments of the present inventions are shown. Indeed, theseinventions may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided by way of example so that this disclosure willsatisfy applicable legal requirements. Like numbers refer to likeelements throughout.

FIG. 1 illustrates main functional components of the persistentadvertisement system, in which 1-101 represents the media contents thatare playing on the terminals. The media contents can be videos, audios,or even images and texts, and they can be in any kind of formats.Component 1-102 represents the terminals and devices, which varies frominternet browsers on PCs, or mobile devices and tablets, to smart TVs.The common features that these terminals and devices are supported inthe persistent advertisement system are 1) able to play media contents,2) able to install customizable software components, 3) network enabled.Therefore, the terminals supported in this part of the system can beinternet browsers with plug-ins, mobile devices or tablets equipped withapplications using dedicated APIs, or smart TVs with specificinterfaces. The functionalities of the software components mentionedabove include 1) extract VDNA fingerprints of the media content whileit's playing, 2) collect required content metadata, 3) collect user'sspecific information.

Actions 1-106 and 1-107 represent the interactions between terminals andthe original media contents that are playing. These two actions arerunning in parallel and independently. 1-106 shows that the terminal isplaying the media content on the user interface. This action reactsaccording to users' operations. Such as play, pause, stop, etc. Themedia player's components used for playing media contents are eitherdedicated applications or plug-in extensible since this softwarecomponent is not only required for media content playback, but also actsas the core functional entity on the client's side. It performs allinteractions between users, devices, clients and servers, among whichaction 1-106 represents interactions between users and the terminaldevices.

1-107 is the action running on the background of the dedicated orplug-in media player component. In other words, it is invisible to theusers and is another route besides action 1-106. The main tasks ofaction 1-107 are 1) extract VDNA fingerprints of the media contents, 2)collect required content metadata and user specific information, such asuser's preferences on the media contents, user's location, terminal typeand so on, 3) send these data for content identification andadvertisement selection, 4) display selected advertisement on theterminal in forms of ad web pages, overlay banners,pre-roll/mid-roll/post-roll ad frames, and so on. All of the tasks aboveare performed in the software component as dedicated applications orplug-ins installed in the terminal devices.

In order to achieve the aforementioned tasks, the software component isrequired to design with permissions to access media contents as well asother user sensitive data, such as user's location, which will beobtained under user authentication. Also the software component iscapable of persisting user preferences over media contents. For example,the genre, authors, albums or other related information about mediacontents, which the specific user is fond of. Such information can becollected by the dedicated software component installed in the terminaldevices, and will be sent to the backend server after the user'sauthentication for analysis, so that the most suitable content-awareadvertisements can be generated for specific users or persistentadvertisements for a group of users sharing targeted interests.

1-108 represents VDNA fingerprints data extracted from the mediacontents by the dedicated software component installed on terminaldevices. VDNA fingerprints are the essence of media contentidentification technology. It extracts the characteristic values of eachframe of image or audio from media contents. This process is similar tocollecting and recording human fingerprints. The fact that VDNAtechnology is entirely based on the media content itself means inbetween media content and generated VDNA is one-to-one mappingrelationship. Compared to the conventional method of using digitalwatermark technology to identify video contents, VDNA technology doesnot require pre-process the video content to embed watermarkinformation. Also, the VDNA extraction algorithm is greatly optimized tobe efficient, fast and lightweight so that it consumes only anacceptable amount of CPU (central processing unit) or memory resourceson the terminal devices. The VDNA extraction process is performed veryefficiently on the terminal side. In addition, extracted fingerprintsare very small in size compared to the media content, which allowstransferring fingerprints over a network possible.

The VDNA extraction algorithm can be various. Take video content forexample: the extraction algorithm can be as simple as the following a)sample the video frame as image b) divide the input image into certainamount of equal sized squares, c) compute average value of the RGB (red,green and blue) values from each pixel in each square, d) in this casethe VDNA fingerprint of this image is the 2 dimensional vector of thevalues from all divided squares. The smaller a square is divided, themore accuracy the fingerprint can achieve. However, at the same time itwill consume more storage. In a more complex version of the VDNAextraction algorithm, other factors such as brightness, alpha value ofthe image, image rotation, clipping or flipping of the screen, or evenaudio fingerprint values will be considered.

The software component on the terminal devices will also collectsmetadata from the media content, which is playing on it (step 1-109).The collected metadata can be obtained either locally from the mediafile such as title, author, album, etc, or collected online throughmedia content websites, where a lot of information regarding the mediacontent can be discovered. For example, audience comments, related mediacontents in the same genre, etc. The software component on the terminaldevices will also send the collected metadata to the identificationserver along with the extracted VDNA fingerprints from 1-108, to helpgenerate more accurate identification results.

The terminal prepares VDNA fingerprints of the media content andcollects metadata of the media content; such as title, author, duration,audience comments and so on. Such data will be sent to the contentidentification server (1-111), which performs content identification andmatching (1-113) against the VDDB (1-112) server where master mediacontents are registered.

The content identification server accepts media content query requestswhich comes along with extracted VDNA fingerprints of the input mediacontents. The input media contents can be any format of audio, video orimage contents. In this case it is processed by dedicated softwarecomponents on the terminal devices, so that a set of VDNA fingerprintsare extracted from the contents. Basically, the content identificationserver is composed by a set of index engines, a set of query engines anda set of master sample databases. All of these components aredistributed and are capable to cooperate with each other.

The index engine or distributed index engines store a key-value mappingwhere the keys are hashed. VDNA fingerprints of the registered mastermedia content and the values are the identifier of the registered mastermedia content. When a query request is triggered, a set of VDNAfingerprints of the input media content is submitted. Then, apre-defined number of VDNA fingerprints are sampled from the submitteddata. The sampled fingerprints are each hashed using the same algorithmas those registered VDNA fingerprints previously hashed, using thesehashed sampled fingerprints to get the values in the registered mapping.Based on statistical research on the matching rates of key framesbetween input media contents and master media contents, it can beconcluded that by being given only a set of sampled fingerprintsextracted from the input media content, it is in highly possible to geta list of candidate matched master content ranked by the hit rate ofsimilarity. The output of index engine will be a list of identifiers ofcandidate media contents ranked by the hit rate of similarity withsampled fingerprints of input media content.

The query engine performs VDNA fingerprint level match between each oneof VDNA fingerprints extracted from input media content and all of theVDNA fingerprints of every candidate media content output from indexengine. There are also scalability requirements for the design of aquery engine to be the same index engine, because the number ofregistered media contents by content owner may vary in differentmagnitude. As a result, the amount of registered VDNA fingerprints canbe massive. In such a condition, distributed query engines are alsorequired to enforce the computing capability of the system. The basicbuilding block of VDNA fingerprint identification algorithm is acalculation and comparison of Hamming Distance of fingerprints betweeninput and master media contents. A score will be given after comparinginput media content with each one of the top ranked media contentsoutputted by the index server. A learning-capable mechanism will thenhelp to decide whether or not the input media content is identified witha reference to the identification score, media metadata, andidentification history.

The result of content identification will be sent together with userspecific information (1-110) collected from the terminals to theadvertisement processor (1-115) for ad selection. The content providesand the advertisement agents will predefine some business rules for thechoice of advertisements. These rules are crucial for generatingcontent-aware persistent advertisements. The terminology ofcontent-aware persistent advertisement is self-explanatory, whereincontent-aware means that the advertisement is selected according to therecognition result of the media content; which comes along from thecontent identification center. The advertisements are also persistent,wherein the persistency can have multiple forms, such as:

-   -   1) geo-location persistency, meaning that users with terminal        devices in a same range of locations should be assigned with        advertisements with specific location hints or information. For        example, promotion of the nearest products which appear in the        recognized media content.    -   2) time persistency, meaning users with terminal devices in the        same duration of time should be assigned with advertisements        with specific date or time hints or information, for example        products on sale on a specific date of a week.    -   3) environment persistency, meaning users with terminal devices        in the same environment should be assigned with advertisements        with specific environmental hints or information, for example        with user's location the backend servers are able to deduce the        weather where the user is located, with such information proper        environmental relevant advertisements can be selected.    -   4) user preferences persistency, meaning users with terminal        devices who share similar preferences on media contents should        be assigned with advertisements with specific preference hints        or information, for example similar artworks can be promoted for        users who are interested in the same kind of genre.

There are more forms of persistency besides the list above, and couldalso be combined together to generate more specific and properadvertisement selections.

The selected advertisements will then be sent back to the terminal(1-118), where the software components installed on it will display theadvertisements in the form provided by the business rules, wherein theforms of advertisements can be static texts, image banners, or eveninteractive animations. The key to successful online advertising is tograb attention without being overly intrusive. Current popular methodsof online advertising include:

-   -   1) banner advertisements: Banner ad online advertising is one of        the first advertising methods used on the internet. Banner        advertisements are also easy to create.    -   2) floating advertisements: This form of online advertising is        seen as intrusive by some people. Floating ads appear to float        or move across a user's screen usually while the media content        is paused.    -   3) interstitial advertisements: This is a relatively new form of        online advertising which has grown in popularity due to online        media contents. Interstitial advertisements are like television        commercials. They usually appear before media content loads or        are placed in some other spot inside the media content.    -   4) pop-under advertising: This is one of the newest and most        controversial forms of online advertising. Unlike pop-up ads,        pop-under ads are sneakier and less intrusive. They don't steal        the attention away from the media content as pop-up ads can do.        Instead, a window will open up in the background with the ad in        it and will be instantly minimized so that the user can continue        to look at the original media content only to find the ad later.        The attractiveness of pop-under advertising is due to its        relatively cheap cost.

Advances in technology also mean advances in online advertising. Morebusinesses have started taking advantage of Streaming Video when itcomes to their online advertising.

FIG. 2 illustrates the workflow of the persistent advertisement systemon the terminal side. When the media is being played, the softwarecomponent checks whether or not the media has been identified before(2-03). If the said media content has been identified before in thisdevice or terminal, it has all the required genuine information aboutthe media. Therefore, it can send the genuine information of the mediaalong with some user authorized data such as user's preferences, user'slocation and terminal type, via internet or mobile networks such asGSM/CDMA (global service for mobile communications) or 3G (thirdgeneration) networks, to start a request to the advertisement selectionprocessor for suitable advertisements (2-09).

When the selected advertisements are returned from advertisementselection server, the terminal then displays these advertisementsaccording to some given or predefined business rules, which can beeither ad webpages, overlay ad banners, pre-roll/mid-roll/post-roll adframes and so on (2-10).

On the other hand, if the playing media content has not been identifiedbefore, the dedicated software component on the terminal will start aroutine for content identification. The routine begins with extractingcharacteristic values from the media content, also known as VDNAfingerprints (2-04). The design of VDNA fingerprints is very compact, sothat it is feasible to transfer over mobile networks. Because someterminals may use mobile networks and they always have lower bandwidth,sending huge amounts of information of the media content to the contentprovide for identification is not realistic, sending only metadata ofthe media content may not result in an accurate identification of thecontent. Therefore, extracting key characteristics of the media contentsand sending the extracted fingerprints of the media contents remits thementioned disadvantages. There are various levels of VDNA extractionalgorithm. Take video content as an example, the extraction algorithmcan be as simple as the following a) sample the video frame as image, b)divide the input image into certain amount of equal sized squares, c)compute average value of the RGB values from each pixel in each square,d) in this case the VDNA fingerprint of this image is the 2 dimensionalvector of the values from all divided squares. The smaller a square isdivided, the more accuracy the fingerprint can achieve, yet at the sametime it will consume more storage. In more complex version of the VDNAextraction algorithm, other factors such as brightness, alpha value ofthe image, image rotation, clipping or flipping of the screen, or evenaudio fingerprint values will be considered.

The terminal also collects metadata of the media content at the sametime, including title, author, and duration of the media content. Suchmetadata information can be helpful to setup some filtration criteriawhen performing VDNA fingerprints matching to shorten the processingtime for identification. The collected metadata can be obtained eitherlocally from the media file such as title, author, album, etc., orcollected online through media content websites, where a lot ofinformation regarding the media content can be discovered, for exampleaudience comments, related media contents in the same genre, etc.

The VDNA fingerprints are also sent to the content identification serverwith the media metadata via internet or mobile networks such as GSM/CDMAor 3G networks. When the terminal receives the genuine information ofthe media content from the content identification server, it will sendthe genuine information of the media along with some user authorizeddata (such as the user's preferences, the user's location and terminaltype) via internet or mobile networks such as GSM/CDMA or 3G networks,to start a request to the advertisement selection processor for suitableadvertisements (2-09). When the selected advertisements are returnedfrom the advertisement selection server, the terminal then displaysthese advertisements according to some given or predefined businessrules, which can be either ad webpages, overlay ad banners,pre-roll/mid-roll/post-roll ad frames and so on (2-10).

FIG. 3 illustrates the workflow of the content identification system.The core processing block of the content identification system is VDDB(3-02). After receiving the VDNA fingerprints and media content metadatafrom the terminal, VDDB starts a quick hash process over the sample VDNAfingerprints with index servers (3-03). The index engine, or distributedindex engines, stores a key-value mapping where the keys are hashed VDNAfingerprints of the registered master media content and the values arethe identifier of the registered master media content. When a queryrequest is triggered, a set of VDNA fingerprints of the input mediacontent is submitted. Then, a pre-defined number of VDNA fingerprintsare sampled from the submitted data. The sampled fingerprints are eachhashed using the same algorithm as the registered VDNA fingerprints werepreviously. Using these hashed sampled fingerprints, you can get thevalues in the registered mapping. Based on the statistical research onthe matching rates of key frames between input media contents and mastermedia contents, it can be concluded that by being given only one set ofsampled fingerprints extracted from the input media content, it is inhighly possible to get a list of candidate matched master content rankedby the hit rate of similarity if all master media contents arefingerprinted and indexed beforehand. This is the optimization ideabehind index servers. Using the index server to pre-process the inputmedia content can save large amounts of processing efforts by avoidinghaving to thoroughly compare all master media contents in detail byquickly generating the best matched media candidate list.

Component 3-04 is the process inside the query engine which performs theVDNA fingerprint level match between each of the VDNA fingerprintsextracted from input media content. All VDNA fingerprints of everycandidate media content output from the index engine. There are alsoscalability requirements for the design of the query engine the sameindex engine. This is because the number of registered media contents bycontent owner may vary in different magnitudes, and as a result theamount of the registered VDNA fingerprints can be massive. In such acondition, distributed query engines are also required to enforcecomputing capability of the system.

The basic building block of the VDNA fingerprint identificationalgorithm is the calculation and comparison of the Hamming Distance offingerprints between the input and master media contents. A score willbe given after comparing the input media content with each one of topranked media contents outputted by the index server (3-05). Alearning-capable mechanism will then help to decide whether or not theinput media content is identified with reference to the identificationscore, media metadata, and identification history. Finally, the resultwill then be sent back to the terminal over the same network.

FIG. 4 illustrates the workflow of the selection of persistentadvertisements in the advertisement processor. The process will receivethe results of the content identification and the user's specificinformation from terminals (4-01 and 4-02). In the case that theterminals are playing the same media contents, the identificationresults should be identical, so that when the advertisement process istrying to get associated advertisements according to the identificationresult, the same set of advertisements are selected for all thoseterminals. Therefore, it is able to achieve the persistency ofadvertisements across multiple terminals and devices, in the sense thatthe advertisements chosen are content-aware from the user's aspect. Theacquirement of associated advertisements (4-03) is based on the mappingof a database defined by content providers and advertisement agents, whosetup sets of mapping criteria and business rules. These rules regardwhat advertisements should be pushed for, what kind of media contentsand in what form the chosen advertisements should be displayed. Thebusiness rules which decide the display form of the chosenadvertisements can include 1) to display a standalone advertisement or aset of alternative advertisements, 2) to display the advertisement asembedded in the webpage playing the media content or a separate webpage,3) to display the advertisements as static texts, images or animation,4) the size and dimensions of the advertisements to display, 5) thepoints of time to display the advertisements while playing the mediacontent can be pre-roll/mid-roll/post-roll advertisement frames. Besidesthose mentioned above, many more business rules customized by contentproviders and advertisement agents can be implemented.

The advertisement processor mainly applies these business rulesaccording to the user specific information acquired by terminals, whereusers have the choices to choose what kind of information to share withthe advertisement processor in order to generate more accurate andspecific advertisements. The options users can be choose to shareinclude 1) the user's preferences which can be predefined by theapplication or the content provider according to the need ofadvertisements or product promotions, 2) the user's locationinformation, users can also choose to share the precision of the currentlocation, 3) the user's terminal type, which allows advertisementprocessor to choose proper advertisements for the terminals or devicesplaying the media content, and so on.

Finally, the proper list of advertisements are generated and transferredback to the terminals for display.

To further understand the details of the present invention, thedefinitions of some processing are necessary which are as follows:

Extract/Generate: to obtain and collect characteristics or fingerprintsof media contents via several extraction algorithms.

Register/Ingest: to register those extracted fingerprints together withextra information of the media content into the database wherefingerprints of master media contents are stored and indexed.

Query/Match/Identify: to identify requested fingerprints of mediacontent by matching from all registered fingerprints of master contentsstored in the database, via advanced and optimized fingerprint matchingalgorithm.

In summary, system and method for providing content-aware persistentadvertisements comprise:

A method for providing content-aware persistent advertisements comprisesthe following steps:

-   -   a) extracting and collecting VDNA (Video DNA) fingerprints of        media contents playing on various network-enabled terminals        including Internet browsers, mobile devices, tablets, smart TVs        (television) and so on, and    -   b) sending the aforementioned extracted fingerprints along with        other information such as metadata, user's location, etc to a        content identification server via Internet or mobile networks.

The aforementioned content-aware means matching and identification ofthe aforementioned media content by content itself instead of metadatainformation such as title, author, etc.

The aforementioned persistent advertisement refers to that when the samemedia content is playing on multiple terminals, it can be identified andrelevant advertisements can be pushed to the aforementioned terminalaccording to identification result.

The aforementioned persistent advertisement refers to media contentrelated advertisements pushed to the aforementioned terminal accordingto the aforementioned identification result, and those chosenadvertisements are mostly similar, but can carry some user specificinformation such as user's location, terminal type or preferences and soon.

The aforementioned media contents can be videos, images, audio, text andany other media type.

The aforementioned extracting and collecting VDNA fingerprints isperformed on the aforementioned terminals where the aforementioned mediacontents are playing.

The aforementioned terminals supported are network-enabled smart devicesincluding PCs (personal computers) running Internet browsers, ordedicated media players, smart mobile devices and tablets, and smart TVs(televisions) and so on.

The aforementioned information sent to the aforementioned contentidentification server along with the aforementioned content fingerprintsare not mandatory, and users have options to decide what information toshare.

The aforementioned extracted fingerprints and other information are sentvia Internet or mobile networks to the aforementioned contentidentification server.

A method for providing content-aware persistent advertisements comprisesthe following steps:

-   -   a) extracting and collecting VDNA (Video DNA) fingerprints of        media contents playing on various network-enabled terminals        including Internet browsers, mobile devices, tablets, smart TVs        (televisions) and so on,    -   b) sending the aforementioned extracted fingerprints along with        other information such as metadata, user's location, etc to        content identification server via Internet or mobile networks,    -   c) selecting accurate advertisements according to the        aforementioned media content and relevant information, and    -   d) finally pushing the aforementioned advertisements back to        user's terminal.

The aforementioned content fingerprints are identified in theaforementioned content identification server using the aforementionedVDNA identification technology, and the result of the aforementionedidentification is considered to be accurate.

The aforementioned VDNA can be extracted from any valid format of theaforementioned media content, and the aforementioned media contentidentification heavily relies on the accuracy and swiftness of theaforementioned VDNA technology.

The aforementioned advertisements are selected according to thecombination of the aforementioned identification of the aforementionedmedia content and other information from users, along with some businessrules given by content provider.

The aforementioned advertisements are pushed to the aforementionedterminals while the aforementioned content is playing.

A system for providing content-aware persistent advertisements comprisesthe following sub-systems:

-   -   a) Sub-system extracting and collecting VDNA (Video DNA)        fingerprints of media contents playing on various        network-enabled terminals including Internet browsers, mobile        devices, tablets, smart TVs (televisions) and so on,    -   b) Sub-system sending the aforementioned extracted fingerprints        along with other information such as metadata, user's location,        etc to content identification server via Internet or mobile        networks,    -   c) Sub-system selecting accurate advertisements according to the        aforementioned media content and relevant information, and    -   d) Sub-system pushing the aforementioned advertisements back to        user's terminal.

The aforementioned terminals supported are network-enabled smart devicesincluding PCs (personal computers) running Internet browsers, ordedicated media players, smart mobile devices, tablets, and smart TVs(televisions) and so on.

The aforementioned terminals supported are equipped with dedicatedcomponents such as plugins for Internet browsers, APIs (ApplicationProgram Interfaces) for mobile devices and interfaces for theaforementioned smart TVs to perform content fingerprint extraction, datacollection and transmission.

The aforementioned network can be used to carry data between theaforementioned terminals and content providers.

The aforementioned advertisements are mainly selected according toresult of content identification, and administrator can associatedifferent the aforementioned advertisements to the same content but pushto different terminal devices according to other information providedtogether by the aforementioned devices.

The method and system of the present invention are based on theproprietary architecture of the aforementioned VDNA® and VDDB®platforms, developed by Vobile, Inc, Santa Clara, Calif.

The method and system of the present invention are not meant to belimited to the aforementioned experiment, and the subsequent specificdescription utilization and explanation of certain characteristicspreviously recited as being characteristics of this experiment are notintended to be limited to such techniques.

Many modifications and other embodiments of the present invention setforth herein will come to mind to one ordinary skilled in the art towhich the present invention pertains having the benefit of the teachingspresented in the foregoing descriptions. Therefore, it is to beunderstood that the present invention is not to be limited to thespecific examples of the embodiments disclosed and that modifications,variations, changes and other embodiments are intended to be includedwithin the scope of the appended claims. Although specific terms areemployed herein, they are used in a generic and descriptive sense onlyand not for purposes of limitation.

1. A method for providing content-aware persistent advertisements, saidmethod comprising: a) extracting and collecting VDNA (Video DNA)fingerprints of media contents playing on various network-enabledterminals including Internet browsers, mobile devices, tablets, smartTVs (television) and so on, and b) sending said extracted fingerprintsalong with other information such as metadata, user's location, etc to acontent identification server via Internet or mobile networks.
 2. Themethod as recited in claim 1, wherein said content-aware means matchingand identification of said media content by content itself instead ofmetadata information such as title, author, etc.
 3. The method asrecited in claim 1, wherein said persistent advertisement refers to thatwhen the same media content is playing on multiple terminals, it can beidentified and relevant advertisements can be pushed to said terminalaccording to identification result.
 4. The method as recited in claim 1,wherein said persistent advertisement refers to media content relatedadvertisements pushed to said terminal according to said identificationresult, and those chosen advertisements are mostly similar, but cancarry some user specific information such as user's location, terminaltype or preferences and so on.
 5. The method as recited in claim 1,wherein said media contents can be videos, images, audio, text and anyother media type.
 6. The method as recited in claim 1, wherein saidextracting and collecting VDNA fingerprints is performed on saidterminals where said media contents are playing.
 7. The method asrecited in claim 1, wherein said terminals supported are network-enabledsmart devices including PCs (personal computers) running Internetbrowsers, or dedicated media players, smart mobile devices and tablets,and smart TVs (televisions) and so on.
 8. The method as recited in claim1, wherein said information sent to said content identification serveralong with said content fingerprints are not mandatory, and users haveoptions to decide what information to share.
 9. The method as recited inclaim 1, wherein said extracted fingerprints and other information aresent via Internet or mobile networks to said content identificationserver.
 10. A method for providing content-aware persistentadvertisements, said method comprising: a) extracting and collectingVDNA (Video DNA) fingerprints of media contents playing on variousnetwork-enabled terminals including Internet browsers, mobile devices,tablets, smart TVs (televisions) and so on, b) sending said extractedfingerprints along with other information such as metadata, user'slocation, etc to content identification server via Internet or mobilenetworks, c) selecting accurate advertisements according to said mediacontent and relevant information, and d) finally pushing saidadvertisements back to user's terminal.
 11. The method as recited inclaim 10, wherein said content fingerprints are identified in saidcontent identification server using said VDNA identification technology,and the result of said identification is considered to be accurate. 12.The method as recited in claim 10, wherein said VDNA can be extractedfrom any valid format of said media content, and said media contentidentification heavily relies on the accuracy and swiftness of said VDNAtechnology.
 13. The method as recited in claim 10, wherein saidadvertisements are selected according to the combination of saididentification of said media content and other information from users,along with some business rules given by content provider.
 14. The methodas recited in claim 10, wherein said advertisements are pushed to saidterminals while said content is playing.
 15. A system for providingcontent-aware persistent advertisements, said system comprising: a)Sub-system extracting and collecting VDNA (Video DNA) fingerprints ofmedia contents playing on various network-enabled terminals includingInternet browsers, mobile devices, tablets, smart TVs (televisions) andso on, b) Sub-system sending said extracted fingerprints along withother information such as metadata, user's location, etc to contentidentification server via Internet or mobile networks, c) Sub-systemselecting accurate advertisements according to said media content andrelevant information, and d) Sub-system pushing said advertisements backto user's terminal.
 16. The system as recited in claim 15, wherein saidterminals supported are network-enabled smart devices including PCs(personal computers) running Internet browsers, or dedicated mediaplayers, smart mobile devices, tablets, and smart TVs (televisions) andso on.
 17. The system as recited in claim 15, wherein said terminalssupported are equipped with dedicated components such as plugins forInternet browsers, APIs (Application Program Interfaces) for mobiledevices and interfaces for said smart TVs to perform content fingerprintextraction, data collection and transmission.
 18. The system as recitedin claim 15, wherein said network can be used to carry data between saidterminals and content providers.
 19. The system as recited in claim 15,wherein said advertisements are mainly selected according to result ofcontent identification, and administrator can associate different saidadvertisements to the same content but push to different terminaldevices according to other information provided together by saiddevices.